EDA HINTS DATA

Machine Learning - Reddit Data

Question : Can we predict if a comment will be cancer related

The goal is to classify Reddit comments into two categories: Cancer-related and Non-Cancer subreddits, to understand if we can predict if a comment is cancer related or no, based on the added source column to label as cancer and non-cancer . The dataset, sourced from the PushShift Reddit dataset, includes subreddit comments from two periods (2021–2023 and 2023–2024).

Methodology

The dataset preparation and preprocessing were crucial to ensuring the models had the appropriate features for learning. The raw data from the PushShift Reddit Dataset included text data and category labels for cancer-related and non-cancer comments.

First, the data was combined into a single DataFrame, with an additional column named source labeling the comments as “Cancer” or “Non-Cancer.” The comments were then preprocessed to ensure they were in a suitable format for the models.

Text tokenization was applied to split the comments into individual words, enabling feature extraction. Tokenization was followed by the removal of stop words, such as “and,” “the,” and “is,” which are frequent but do not carry significant information for classification. The tokenized words were then converted into numerical vectors using CountVectorizer, which captures word frequencies. To enhance the importance of rare but meaningful words, TF-IDF (Term Frequency-Inverse Document Frequency) scores were computed. This step ensured that commonly occurring but uninformative words received less weight, while rare and distinctive words were emphasized.

Finally, the labels (“Cancer” and “Non-Cancer”) were encoded numerically as 0 and 1, respectively, using StringIndexer. These preprocessing steps were implemented as a PySpark pipeline to ensure consistency and scalability.

The processed dataset was then split into 80% training and 20% testing subsets. Two machine learning models, Naive Bayes and Logistic Regression, were trained on the training data using PySpark’s built-in implementations. The evaluation metrics, such as accuracy, precision, recall, F1-score, and AUC-ROC, were computed on the test set. Additionally, confusion matrices were generated to visualize the distribution of correct and incorrect predictions for both models.

Logistic Regression and Naive Bayes were chosen for their simplicity, efficiency, and effectiveness in text classification. Naive Bayes handles high-dimensional data well with its independence assumption, while Logistic Regression provides robust, interpretable probability predictions. These models serve as strong benchmarks for binary classification tasks.

---

Naive Bayes

Conclusion

Understanding whether a comment is cancer-related is crucial for enhancing online support systems and health communication. Cancer-related subreddits often serve as platforms for individuals seeking emotional support, sharing personal experiences, or discussing treatment options. Automatically identifying such comments can help moderators curate relevant content, ensure timely responses to critical queries, and provide researchers with valuable insights into public sentiment, challenges, and trends related to cancer. This understanding is particularly important for designing interventions, improving healthcare accessibility, and fostering a supportive community for those affected by cancer.